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ABSTRACT 

Program invariants can help software developers identify 
program properties that must be preserved as the software 
evolves, however, formulating correct invariants can be chal- 
lenging. In this work, we introduce iDiscovery , a technique 
which leverages symbolic execution to improve the quality 
of dynamically discovered invariants computed by Daikon. 
Candidate invariants generated by Daikon are synthesized 
into assertions and instrumented onto the program. The in- 
strumented code is executed symbolically to generate new 
test cases that are fed back to Daikon to help further refine 
the set of candidate invariants. This feedback loop is exe- 
cuted until a fix-point is reached. To mitigate the cost of 
symbolic execution, we present optimizations to prune the 
symbolic state space and to reduce the complexity of the 
generated path conditions. We also leverage recent advances 
in constraint solution reuse techniques to avoid computing 
results for the same constraints across iterations. Experi- 
mental results show that iDiscovery converges to a set of 
higher quality invariants compared to the initial set of can- 
didate invariants in a small number of iterations. 

Categories and Subject Descriptors 

D.2.4 [Software Engineering]: Software/Program Verifi- 
cation; D.2.5 [Software Engineering]: Testing and De- 
bugging 

General Terms 

Algorithms, Verification, Experimentation 

Keywords 

Invariant generation, Symbolic execution, Model checking 

1. INTRODUCTION 

While researchers have long recognized the value of doc- 
umenting properties of code, e.g., as behavioral specifica- 
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tions [20,24], in the development and maintenance of cor- 
rect software systems, writing specifications for non-trivial 
functional correctness properties is quite challenging and re- 
mains a largely manual process. Consequently, it is rare to 
find software systems with accompanying specifications that 
are up-to-date. This creates challenges not only for reason- 
ing about correctness, but it also hinders the development of 
effective testing and analysis techniques that leverage spec- 
ifications. An example of the latter — in our recent work on 
developing an incremental analysis to support software evo- 
lution [36], we were not able to find any existing artifacts 
that included multiple versions of expected properties cor- 
responding to the different versions of code as it evolved. 

Techniques to automatically extract properties of code, 
say to create likely specifications, provide users with an ini- 
tial set of specifications from which additional, or more pre- 
cise, specifications can be derived. Such techniques have 
been investigated by a number of researchers over the last 
several decades using a variety of static [5,8,22,32] and dy- 
namic [9, 12] techniques. While some of these techniques 
are efficient and applicable to real-world programs, the gen- 
erated properties may not accurately characterize the pro- 
gram’s behavior [28,31]: (1) they may contain many incor- 
rect or imprecise properties; and (2) they may miss many 
valid properties. The focus of this paper is on extracting 
higher quality properties by addressing both of these issues. 

Dynamic invariant discovery is a well-known approach for 
generating invariants. It was pioneered over a decade ago 
by Daikon [12] and provided the foundation for a number 
of subsequent techniques [1,3,23]. Dynamic invariant dis- 
covery has a simple and practical basis: execute a program 
against a given test suite, monitor the executions at control- 
points of interest to generate program traces, and check the 
program states collected in the traces to validate a set of 
invariants to see which ones hold and compute candidate 
program invariants [12]. The number of the invariants in- 
ferred by these tools and their correctness is often highly 
dependent on the quality of the test suites used [12,28]. 
We are unaware of any current work on how to construct 
a test-suite that would enable dynamic invariant inference 
techniques to generate better program invariants. 

In this work, we introduce iDiscovery , a technique that 
employs a feedback loop [35] between symbolic execution [6, 
19] and dynamic invariant discovery to infer more accurate 
and complete invariants until a fix-point is reached. Our em- 
bodiment of iDiscovery uses Daikon as the dynamic invariant 
inference technique. In each iteration, iDiscovery transforms 
candidate invariants inferred by Daikon into assertions that 
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are instrumented in the program. The instrumented pro- 
gram is analyzed with symbolic execution to generate ad- 
ditional tests to augment the initial test suite provided to 
Daikon. The key intuition behind iDiscovery is that the 
constraints generated on the synthesized assertions provide 
additional test inputs that can refute incorrect /imprecise in- 
variants or expose new invariants. Therefore, when the new 
inputs are used to augment the previous test suite, dynamic 
invariant discovery will be based on a richer set of program 
executions enabling discovery of higher quality invariants. 

A key element in the performance cost of iDiscovery is 
the high cost of symbolic execution. To mitigate the cost 
of symbolic execution, iDiscovery provides two optimiza- 
tions: assertion separation and violation restriction. As- 
sertion separation leverages the underlying search engines’ 
non-deterministic choice and backtracking features to focus 
symbolic execution on checking one assertion at a time. This 
generates fewer path conditions that are overall less complex 
and in turn reduces the constraint solving time. Violation 
restriction uses the state of the underlying execution engine 
to generate at most one violation of each assertion, since one 
test that reveals the violation is sufficient to form a counter- 
example. Both of these configurations can be applied in 
tandem to amortize the cost of iDiscovery. Moreover, iDis- 
covery’s feedback loop, which repeatedly uses symbolic ex- 
ecution, involves iteratively checking the same code against 
different sets of properties. Thus, our approach offers a 
substantial opportunity to memoise and re-use the results 
of symbolic execution from previous iterations to further re- 
duce its cost [4,33,37]. Our iDiscovery tool uses the Green 
library [33] for the Symbolic PathFinder [26] to re-use con- 
straint solving results, which together with assertion separa- 
tion and violation restriction, minimize the overall symbolic 
execution cost for improving dynamic invariant discovery. 

We make the following contributions: 

• Dynamic invariant discovery and symbolic ex- 
ecution. We combine Daikon and symbolic execution 
to generate test inputs that allow Daikon to discover 
higher quality invariants. 

• iDiscovery. We present our core technique iDiscov- 
ery, which embodies our idea to dynamically discover 
invariants for Java programs. 

• Optimizations. We present two optimizations: (a) 
assertion separation and (b) violation restriction that 
enhance the efficiency of iDiscovery. 

• Evaluation. We present an experimental evaluation 
using a suite of Java programs that have been used 
in previous work on invariant discovery and symbolic 
execution. The experimental results show that iDis- 
covery converges to a set of higher quality invariants 
than the initial set of invariants computed by Daikon 
in a small number of iterations. 

2. MOTIVATING EXAMPLE 

We demonstrate our feedback-driven technique, iDiscov- 
ery, for inferring and refining program invariants using the 
example in Fig. 1. In Fig. 1 we first present snippets of code 
from the Wheel Brake System (WBS) [29] and the tests used 
to generate the initial invariants. We then show the arti- 
facts produced by our technique: 1) the assertions generated 


public class WBS { 

BSCU_SystemModeSelCmd_rlt_PRE = 0; 

Nor_Pressure = 0; 

public void update (int PedalPos, boolean AutoBrake, 
boolean Skid){ 

rlt_PRE2 = 100; 

if (PedalPos == 0) 

BSCU_Command_PedalCommand_Switchl = 0; 
else if (PedalPos == 1) 

BSCU_Command_PedalCommand_Switchl = 1; 
else if (PedalPos == 2) 

BSCU_Command_PedalCommand_Switchl = 2; 
else if (PedalPos == 3) 

BSCU_Command_PedalCommand_Switchl = 3; 
else if (PedalPos == 4) 

BSCU_Command_PedalCommand_Switchl = 4; 
else BSCU_Command_PedalCommand_Switchl = 0; 


/** Subset 

of 

the initial 

test inputs 

**/ 

PedalPos 

= 

0, 

AutoBrake 

= 

false , 

Skid 

= 

false; 

PedalPos 

= 

1, 

AutoBrake 

= 

false , 

Skid 

= 

false; 

PedalPos 

= 

2, 

AutoBrake 

= 

false , 

Skid 

= 

false ; 

PedalPos 

= 

3, 

AutoBrake 

= 

false , 

Skid 

= 

false ; 

PedalPos 

= 

4, 

AutoBrake 

= 

false , 

Skid 

= 

false ; 

PedalPos 

= 

5, 

AutoBrake 

= 

false , 

Skid 

= 

false; 


/** Assertions generated from initial invariants **/ 
Al: assert (this. rlt_PRE2 > PedalPos); 

A2 : assert (PedalPos >= 0); 

A3: assert (this .BSCU_SystemModeSelCmd_rlt_PRE <= 
PedalPos) ; 

A4 : assert (this. rlt_PRE2 == 100); 

A5 : assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . Nor_Pressure) ; 

A6: assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . BSCU_rlt_PREl) ; 


/** New tests generated in the first iteration **/ 
PedalPos = 100, AutoBrake = false, Skid = false; 
PedalPos = -1000, AutoBrake = false, Skid = false; 


/** Assertions updated in the first refinement **/ 
A4 : assert (this. rlt_PRE2 == 100); 

A5 : assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . Nor_Pressure) ; 

A6: assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . BSCU_rlt_PREl) ; 

A7 : assert (this. rlt_PRE2 >= PedalPos); 


/** New test generated in the second iteration **/ 
PedalPos = 101, AutoBrake = false, Skid = false; 


/** Assertion updated in the second refinement **/ 

A4 : assert (this. rlt_PRE2 == 100); 

A5: assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . Nor_Pressure) ; 

A6 : assert (this .BSCU_SystemModeSelCmd_rlt_PRE == 
this . BSCU_rlt_PREl) ; 

Figure 1: Parts of the WBS example, the test cases 
generated by symbolic execution at each step, and 
the assertions generated from the Daikon invariants. 

based on the Daikon invariants, 2) the new tests generated 
by symbolic execution at each iteration of our technique, 
and 3) the updated assertions based on the program invari- 
ants generated by Daikon at each iteration of iDiscovery. 
The assertions and tests shown in Fig. 1 are a subset of the 
assertions and tests generated by iDiscovery for the WBS 
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program. The complete results for the WBS example are 
discussed in Section 5. 

The WBS program contains an update method with three 
input variables: PedalPos, AutoBrake, and Skid. These in- 
put variables are treated as symbolic values while all other 
fields in the systems are initialized with concrete values. An 
initial set of test inputs is generated for the update method 
using symbolic execution. A subset of these tests is shown 
in Fig. 1; in these tests there are six possible values for the 
input parameter PedalPos: 0, 1, 2, 3, 4, 5 while the Au- 
toBrake and Skid variables are set to false. The six input 
values of PedalPos cover all of the branches shown in the 
update method in Fig. 1. Daikon monitors the concrete ex- 
ecution traces of the WBS program that are generated by 
the initial test inputs to infer the candidate invariants re- 
lated to the PedalPos input variable and other fields in the 
WBS program. These invariants are transformed into Java 
assertions A 1 through A6 as shown in Fig. 1. 

Symbolic execution is performed on the original program 
instrumented with assertions A1-A6. Symbolic execution of 
the object code (i.e. Java bytecode) corresponding to the 
assertions creates additional constraints on the path condi- 
tions of the program. The SMT solver now generates new 
test inputs for the program by solving these path conditions. 
For example, the constraint PedalPos < 0 is generated by 
symbolic execution along the false branch of the conditional 
branch for assertion A2: PedalPos > 0. The SMT solver 
provides a solution of —1000 for PedalPos based on this new 
constraint; this in turn results in a new test input. Similarly, 
the constraints this .rlt_PRE2 == 100 A this .rlt_PRE2 < 
PedalPos are generated by symbolic execution along the true 
branch of assertion A4 and the false branch of assertion Al. 
The SMT solver generates a solution 100 for PedalPos to 
satisfy these constraints, which is used to create another 
new test input value. 

The values of —1000 and 100 are provided as test inputs 
for the PedalPos input variable while the AutoBrake and 
Skid are set to false. Using the test traces generated by 
these new test inputs, Daikon refines the set of candidate 
invariants. Note that the test traces are generated based 
the original program without the assertions generated from 
invariants. Using information from the new traces, Daikon 
deletes (does not generate) assertions Al, A2, and A3; re- 
tains A4, A5, A6; and adds AT. 

In the second iteration of iDiscovery, the invariants are 
again transformed into Java assertions and instrumented on 
the original program. Symbolic execution generates new 
constraints, this .rlt_PRE2 == 100 A this . rlt_PRE2 < 
PedalPos, for assertions A4 (true branch) and AT (false 
branch). These constraints cause the SMT solver to com- 
pute a solution of 101 for the PedalPos variable. A new test 
input is created with this value for PedalPos, and AutoBrake 
and Skid are set to false. Daikon then refines the candidate 
invariants using the program trace for the new input. In 
the second iteration, Daikon deletes assertion AT and does 
not add any new assertions. For the code-snippet shown 
in Fig. 1 this represents a fix-point. For this example, iDis- 
covery successfully refutes 3 incorrect /imprecise invariants 
computed by Daikon (50% of the initial invariants). 

3. APPROACH 

In this section, we present the details of our iDiscovery ap- 
proach. Given a program and an initial test suite, iDiscovery 



Figure 2: iDiscovery Overview. 


automatically and iteratively applies two techniques: 1) dy- 
namic invariant inference, e.g., Daikon [12], and 2) symbolic 
execution [6,19], to infer program invariants for Java pro- 
grams. An overview of the iDiscovery technique is shown in 
Figure 2. iDiscovery transforms the program invariants in- 
ferred by Daikon into Java expressions within assert state- 
ments. All of the paths in the original program code and 
the synthesized assertions are explored using symbolic exe- 
cution in an attempt to generate additional test inputs. For 
every assertion, constraints encoding the true and the false 
branches of the assertions are added to the path condition 
computed by symbolic execution. The resulting path con- 
ditions are solved using an off-the-shelf decision procedure, 
in an attempt to generate additional test inputs. The ad- 
ditional test inputs, together with the initial test suite, are 
then provided to Daikon in the next iteration of iDiscov- 
ery. Note that iDiscovery makes no modifications to either 
the invariant inference technique or the symbolic execution 
algorithm; each is treated as a black-box by iDiscovery. 

During each iteration of iDiscovery, the set of inferred 
invariants is refined until a fix-point is reached, e.g., the in- 
ferred invariants are unchanged with respect to the previous 
iteration of the algorithm. An invariant is refuted by Daikon 
when a counter example is found. Therefore, theoretically, 
all the invariants refuted in each iteration of iDiscovery are 
either incorrect or imprecise, because the symbolic execution 
engine of iDiscovery is able to generate counter examples for 
those invariants. New invariants can also be augmented in 
each iteration of iDiscovery because more evidence support 
can be found by the additional tests generated by iDiscov- 
ery. Note that similar to other dynamic invariant discovery 
techniques [9,12,16], the invariants discovered by iDiscov- 
ery may be unsound or incomplete (due to the limitations 
of Daikon). However, iDiscovery can generate valuable tests 
to guide Daikon to refute incorrect or imprecise invariants, 
or augment un-observed invariants. 

Formally, the overall iDiscovery algorithm is shown in 
Algorithm. 1. The inputs to the algorithm are the pro- 
gram under analysis V, and an initial test suite Tinit • The 
set of inferred invariants is initialized to 0 at line 2. The 
set of execution traces used by the invariant inference al- 
gorithm, iNVARlNFER(r), is computed by invoking function 
TraceExec(V, Tinit) at line 3. Recall that iDiscovery treats 
the invariant inference algorithm as a black box, providing 
only the program and the tests as input. During each itera- 
tion of the loop starting at line 4, the iDiscovery algorithm 
first invokes the invariant inference algorithm to generate a 
set of candidate invariants. If the set of inferred invariants 
is equivalent to the previous set of inferred invariants, iDis- 
covery has reached a fix-point and exits the loop at line 8. 
If the set of candidate invariants is changed, then at line 9, 
V is instrumented with code to check the invariants, e.g., 
with assert statements, and at line 10 symbolic execution 
of the instrumented version of V is performed. The test 
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Algorithm 1: iDiscovery Algorithm 


Input: Program V, test suite T n it 
Output: Invariants X 

1 begin 


2 


3 

4 

5 


6 


// Initialize the invariant set as empty 

X <- 0 

// Record the execution trace for the original 
test suite 

T TRACEEXEc(X,7imt) 
while true do 
Zold T 

// Infer invariants based on the test 
execution traces 

X iNVARlNFER(r) 

// Terminate the algorithm if the fix-point is 
reached 


r if X = X oid then 

8 |_ break 


9 


10 


li 


// Annotate program under test with property 
checks 

V' •<— PropertyInstr(7 :? ,X) 

// Use symbolic execution to generate 
additional test inputs 

Tsym SYMBCEXEC('P / ) 

// Expand the test traces with the generated 
tests 

T <— TRACEEXEC(7 :? , Tsym ) U T 


12 


return X// Return the final set of inferred 
invariants 


inputs computed by symbolic execution are used to aug- 
ment the set of execution traces used by the invariant in- 
ference algorithm in the next iteration of the loop. iDiscov- 
ery includes test inputs for all paths explored by symbolic 
execution (non- violating and counter-examples). When a 
fix-point is reached, the loop terminates and the inferred 
invariants are returned at line 12. 

3.1 Complexity Analysis 

Symbolic execution is a powerful analysis technique; how- 
ever, scalability is an issue because of the large number 
of execution paths generated during the symbolic analysis. 
This is known as the path explosion problem, and is fur- 
ther exacerbated in iDiscovery by the addition of the as- 
sert statements to the code. Symbolic execution generates 
a set of path conditions, 4>, for the program under test that 
does not contain any synthesized assertions from invariants. 
During the symbolic execution in iDiscovery, a path condi- 
tion, 0 G <£>, for the original program is further expanded by 
adding constraints from the synthesized assertions. 

Suppose iDiscovery instruments m synthesized assertions 
in the locations specified by Daikon (e.g., beginning of pro- 
gram methods, or at the end of program methods). For the 
ease of presentation, assume the synthesized assertions are 
not contained within a loop, then for each path condition, 
4> G 4>, there can be at most ra+ 1 potential path conditions 
generated in iDiscovery (when 4> reaches all the m asser- 
tions). Note that iDiscovery is applicable even when synthe- 
sized assertions are contained in loops; in that case, the total 
number of expanded paths for 4> can be £i<i<m|7</>( a i)l + 1> 
where | 7 </>(ai)| > 1 denotes the number of instance asser- 
tions unrolled from loops for ai in path 0 (^(a*) | = 1 for 
assertions not in loops). The set of path conditions con- 
taining constraints from 4> and the synthesized assertions is 


described as {fa A (Ai<j<fc-i a i) A -, afc|l < k < m + 1}; 
where each ai for 1 < i < m is the predicate constraint ex- 
tracted from the corresponding assert statement, each fa 
for 1 < i < m is a prefix of the path condition 0 that reaches 
a,i , and (/>m + 1 = </>Aa m +i = false so that the corresponding 
expanded path specifies the path that satisfies all assertions 
when k == m + 1. To illustrate, Fig. 3(a) shows an execu- 
tion path annotated with the path condition </> as well as 
the synthesized assertions. In Fig. 3, white nodes represent 
constraints in the original program, the black nodes denote 
the synthesized assertions, and the square nodes denote the 
program exit points. The path conditions generated due to 
the synthesized assertions are annotated at the correspond- 
ing exit nodes. In this way, the original program path 0 
is expanded into 3+1=4 different paths after instrumented 
with 3 assertions. Note that <fi is true because a\ is instru- 
mented at the beginning of the program. In total, symbolic 
execution of the program instrumented with synthesized as- 
sertions can generate up to |<f>| * (m + 1) path conditions. 

3.2 Cost Reduction 

The path explosion problem in iDiscovery can be divided 
into two dimensions: (1) checking assertions along each orig- 
inal program path can be expensive when there are many 
assertions because the constraints accumulate over all the 
assertions, (2) the same assertions are repeatedly checked by 
different program paths. We discuss two optimizations that 
allow us to mitigate the path explosion problem in the above 
two dimensions respectively: (1) assertion separation where 
we direct symbolic execution to check one instrumented as- 
sertion in isolation at a time for each original program path, 
and (b) violation restriction to cause symbolic execution 
to generate at most one violation of each instrumented as- 
sertion across different program paths. Figure 3(b) shows 
the set of expanded paths explored for the original path </> 
during symbolic execution in iDiscovery without assertion 
separation in the second column and with assertion separa- 
tion in the third column. As shown in the second column, 
each path condition contains constraints that represent the 
true branches of assertions (satisfied) with a constraint from 
the false branch of an assertion (refuted). The reason is 
that symbolic execution will treat assertions equally with 
branches. By simply checking constraints in the third col- 
umn we can reduce the cost of symbolic execution. In ad- 
dition, symbolic execution in the default configuration of 
iDiscovery attempts to find all possible violations of an as- 
sertion, since it can appear in multiple paths, each of which 
will try to violate the assertion. In order to mitigate this 
cost we configure iDiscovery to generate test inputs for only 
the first violation of an assertion. In the violation restric- 
tion configuration, iDiscovery keeps track of assertions that 
have been violated since the start of symbolic execution, and 
prunes paths leading to these violated assertions. 

3.3 Discussion 

The two optimizations presented in this work reduce the 
number of path conditions generated in iDiscovery, however, 
they may also cause iDiscovery to generate fewer additional 
test inputs. The optimizations enable the analysis of arti- 
facts that are too large to be analyzed with the standard 
iDiscovery algorithm. Therefore, in Section 5, we present a 
detailed study to evaluate the optimizations for both effec- 
tiveness and efficiency on various subject systems. Further- 
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02 A ai A ->a 2 /^& 2 
*a 2 
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03 — &1 A &2 A &3 
0 = &1 A &2 A &3 


Step 

No Separation 

Separation 

1 

01 A -<ai 

01 A — >01 

2 

02 A ai A ->a2 

02 A — >02 

3 

03 A Oi A 02 A -103 

03 A ->03 

m 

0 m A Oi A 02 A ... A 0777,-1 A —>0777 

0m A — >Ot77 

m + 1 

0 A 01 A 02 A ... A o m _i A a 777 

0 


Figure 3: (a) Example path expanded with assertions (b) Assertion checking 


more, various techniques have been developed to reduce the 
cost of symbolic execution by reusing constraint solutions, 
e.g., [33,37]. Given the iterative nature of the iDiscovery 
algorithm, it is an ideal candidate for constraint re-use solu- 
tions. Hence, in this work we leverage an existing constraint 
solution reuse technique, Green [33] to further mitigate the 
cost of symbolic execution in iDiscovery. 

iDiscovery is guaranteed to terminate if the underlying dy- 
namic invariant inference technique satisfies two key prop- 
erties: (a) the invariant inference is deterministic, i.e. , the 
same invariants are generated for a given test suite each time 
and (b) a potential invariant, violated by at least one test 
in the given suite, is not generated as a candidate invariant. 
In each iteration of iDiscovery, the set of invariants inferred 
may increase or decrease. A refuted invariant, however, is 
never generated again by Daikon [11]. Furthermore, there 
is an upper bound to the possible candidate invariants that 
Daikon can generate for a given program. These elements 
lead us to believe that iDiscovery when used with Daikon 
will terminate for most programs (Also confirmed by our 
experimental evaluation in Section 5). 

4. DETAILED OPTIMIZATION 

To evaluate our technique we implement iDiscovery using 
Daikon to dynamically infer program invariants and Sym- 
bolic PathFinder (SPF) to perform symbolic execution. iDis- 
covery takes advantage of SPF’s support for non-deterministic 
choices within its underlying engine (Java PathFinder — JPF) 
and leverages the Green extension for reusing constraint so- 
lutions. We use CVC3 and CORAL as the underlying solvers 
to check satisfiability of path conditions and generate so- 
lutions. The main code for coordinating and combining 
Daikon and SPF is implemented using bash scripts. The 
instrumentation of Daikon invariants into the program un- 
der test is implemented using the Eclipse JDT framework. 

To implement the assertion separation optimization, iDis- 
covery automatically generates a wrapper for each assertion 
as shown in Figure 4(a). Each synthesized assertion, a^, is 
guarded by Verify .getBooleanO which is a modeling prim- 
itive in JPF that non-deterministically generates both true 
and false choices during symbolic execution. When Ver- 
ify .getBooleanO returns false, assertion a* is not checked 
and the search continues executing the instruction that fol- 
lows the if statement. When Verify .getBooleanO returns 
true , assertion ai is checked. The Verify . ignorelf (true) is 
another modeling primitive in JPF, which forces the search 
to backtrack. Here it is used to force the search to backtrack 
after checking ai. The combination of these two modeling 
primitives enable iDiscovery to check each assertion sepa- 
rately as shown in the Separation column of Figure 3(b). 
Consider the example in Figure 4(b), where the dashed nodes 
represent the new choices from Verify .getBooleanO that 


control whether to check an assertion. The symbolic execu- 
tion engine generates and explores non-deterministic choices. 
The Verify . ignore (true) statement forces symbolic exe- 
cution to backtrack to the previous choice point when the 
corresponding assertion has been checked. The pruned parts 
of the search tree are enclosed in dashed lines for each asser- 
tion. The final set of paths explored by symbolic execution 
are annotated with the path conditions at the correspond- 
ing exit nodes. Note that the modeling primitives are not 
considered as path conditions and the final path conditions 
generated are exactly the same as the ones shown in the 
Separation column of Figure 3(b). 

To restrict the number of times an assertion is violated 
to at most once, iDiscovery also uses modeling primitives in 
JPF to wrap assertions and record the assertion checking 
information. For each assert statement, iDiscovery auto- 
matically creates a wrapper as shown below: 

if (Verify. getBooleanO ){ //non-deterministic boolean 
choice 

if (Verify .getCounter (i) < 1 && !(oD) f 
Verify . incrementCounter (i) ; 

throw new AssertionError ( ‘ c ith assertion failed’’); 

> 

Verify . ignorelf (true) ; / /backtrack 

> 


The code, Verify. getCounter (i) , controls whether asser- 
tion ai is checked (1 < i < m and m is the total number 
of synthesized assertions). When an assertion is violated 
on some execution path, the counter for that assertion is 
increased by 1, indicating the assertion has been violated 
during the search and does not need to be checked again. In 
the best case scenario, if the first path violates all the asser- 
tions then the pruning can reduce the number of assertion 
checks from |4>| * (m + 1) to |4>| +m. The reason is that only 
the first path that reaches the assertions will be explored 
to + 1 times for all to assertions, while the other |4>| — 1 
paths will only be checked once each. In the worst case sce- 
nario, if none of the paths lead to an assertion violation then 
no paths will be pruned. 

5. EVALUATION 
5.1 Artifacts 

We evaluate iDiscovery on four Java artifacts. 1. Traffic 
Anti- Collision Avoidance System (TCAS) is an aircraft col- 
lision avoidance system consisting of one class and approx- 
imately 150 lines of code. We use three randomly selected 
versions of TCAS from the Software Infrastructure Repos- 
itory (SIR) [30]. We inline all functions in TCAS into the 
TCAS . alt_sep_test method, and apply iDiscovery on that 
method. 2. Wheel Brake System (WBS) is a synchronous 
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//non-deterministic boolean 
choice 

if (Verify .getBooleanO ){ 
//assertion checking 
assert (aQ ; 

//backtrack 
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Figure 4: (a) Assertion separation wrapper (b) Assertion separation backtrack tree 


reactive component derived from the WBS case example 
found in ARP 4761 [29]. It consists of a single class and 
231 lines of code. We inline all functions in WBS into the 
WBS. update method, and apply iDiscovery on that method. 
3. Altitude Switch (ASW) is a synchronous reactive com- 
ponent from the avionics domain. The ASW consists of a 
single class and 610 lines of code. We apply iDiscovery on 
the asw.MainO method, which implements the main func- 
tionality in ASW. 4- The Apollo Lunar Autopilot (Apollo) 
is a model created by an engineer from the Apollo Lunar 
Module digital autopilot team. We apply iDiscovery on the 
rjc .MainSymbolic method, which is the main method that 
invokes all other methods in Apollo. Apollo contains 2.6 
KLOC in 54 classes. 

5.2 Experimental Setup 

5 . 2 . 1 Independent Variables 

The independent variables in our study are as follows: 

I VI: Different iDiscovery Optimizations. We evalu- 
ate four possible configurations of iDiscovery: (a) iDiscovery 
with no optimizations, (b) iDiscovery with only assertion 
separation (Optimization- 1), (c) iDiscovery with only viola- 
tion restriction (Optimization-2), (d) iDiscovery with both 
optimizations. 

IV2: Different Solver Configurations. As iDiscovery 
applies symbolic execution to similar programs, i.e. , the 
same code with different synthesized assertions, at each iter- 
ation, the solver’s opportunities for constraint solution reuse 
may affect the efficiency of iDiscovery significantly. We eval- 
uate iDiscovery with and without the Green solver inter- 
face [33] to measure the benefits of constraint reuse. 

IV3: Different Initial Test Suites. The initial test suite 
provided to iDiscovery may affect the invariants computed 
by iDiscovery. We investigate the impact of different initial 
test suites on iDiscovery: (a) an initial test suite generated 
by full symbolic execution, (b) an initial test suite with the 
same size as the test suite generated by symbolic execution 
but generated using a random test generator, (c) initial test 
suites of varying sizes (e.g., 5%, 10%, 15%, and 20%) of 
randomly generated tests. 

We set a time-out limit of 20 hours for each of our exper- 
imental settings. All of the experiments were performed on 
a Dell machine with an Intel i7 Quad-Core processor, 8G 
RAM, and Ubuntu 12.04 64-bit version OS. 

5 . 2.2 Dependent Variables 

DV1: Effectiveness. For each configuration of iDiscov- 
ery, we collect the number of refuted invariants (indicating 
iDiscovery’s effectiveness in falsifying incorrect or imprecise 
invariants), and added invariants (indicating iDiscovery’s ef- 


fectiveness at augmenting new invariants). 

DV2: Efficiency. For each configuration of iDiscovery, we 
collect the time costs for test trace collection, invariant infer- 
ence, and test augmentation time using symbolic execution 
at each iteration. 

5.3 Results and Analysis 

Table 1 shows the detailed experimental results for the 
iDiscovery configuration without optimizations using an ini- 
tial test suite generated by symbolic execution. The first 
column lists the artifact names. The second column lists 
the iDiscovery iteration. For each iteration, column 3 lists 
the number of invariants generated by Daikon, and columns 
4 and 5 list the number of deleted and augmented invariants 
respectively, with respect to the original set of invariants 
generated by Daikon in the first iteration. Columns 6 and 
7 list the execution time costs (in seconds) for computing 
the execution traces and the invariant inference time respec- 
tively, for each iteration. Columns 8 to 12 list the number of 
test cases generated by symbolic execution and the cost of 
symbolic execution to analyze the code instrumented with 
the invariants generated in each iteration: the number of 
solvers calls, the number of backtracked states, the max- 
imum memory consumption, and the symbolic execution 
time 1 . We only show the iterations up to the point where 
iDiscovery reaches a fix-point. Similarly, Table 2 shows the 
detailed results for iDiscovery using both optimizations (as- 
sertion separation and violation restriction) with the initial 
test suite generated by symbolic execution. 

The results in Table 1 show that for the TCAS and WBS 
artifacts, iDiscovery is able to refine invariants effectively 
without any optimizations. For the ASW and Apollo ar- 
tifacts, however, iDiscovery times out after 20 hours. For 
the first version of TCAS, iDiscovery successfully falsified 
71.50% of the original invariants (148 out of 207), indicating 
that the additional test cases generated by symbolic execu- 
tion helped Daikon falsify invariants. Theoretically, Daikon 
falsifies invariants only when it has counter-example(s), in- 
dicating that all falsified invariants are correctly deleted. 
In addition, the final set of augmented invariants are quite 
small in number (e.g., only 2 for Apollo, and 0 for all other 
subjects). The reason is that the initial test suites gener- 
ated by symbolic execution already give enough evidence 
support for the possible candidate invariants, making iDis- 
covery mainly refute invariants in this case (also confirmed 
by our later study on different initial test suites). 

iDiscovery reaches a fix-point in a relatively small num- 
ber of iterations. In the subjects where iDiscovery termi- 

x The symbolic execution time for WBS is sometimes 0 be- 
cause the time is too small to be captured by the SPF timer. 
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Table 1: Experimental results for iDiscovery without optimizations 


Subjects 

Iter. 

j Invariants 

| Invariant Cost 

i Symbolic Execution Cost 



Num 

Del 

New 

Tracing 

Inference 

Tests 

SCalls 

States 

Memory 

Time 

TCAS-vl 

1 

207 

0 

0 

40s 

11s 

183 

10066 

10067 

129MB 

250s 


2 

79 

132 

4 

109s 

13s 

193 

2894 

2895 

129MB 

44s 


3 

63 

144 

0 

115s 

15s 

74 

2342 

2343 

118MB 

28s 


4 

59 

148 

0 

44s 

16s 

72 

2090 

2091 

118MB 

22s 

TCAS-v2 

1 

149 

0 

0 

100s 

13s 

445 

13944 

13945 

126MB 

348s 


2 

78 

77 

6 

265s 

17s 

299 

5604 

5605 

238MB 

93s 


3 

63 

86 

0 

178s 

20s 

167 

5404 

5405 

128MB 

64s 


4 

59 

90 

0 

99s 

22s 

165 

4752 

4753 

127MB 

50s 

TCAS-v3 

1 

149 

0 

0 

218s 

16s 

483 

14804 

14805 

128MB 

356s 


2 

78 

77 

6 

285s 

20s 

321 

5830 

5831 

128MB 

87s 


3 

63 

86 

0 

189s 

23s 

184 

5750 

5751 

127MB 

63s 


4 

59 

90 

0 

109s 

25s 

182 

5022 

5023 

126MB 

46s 

WBS 

1 




11s 

2s 

26 

100 

101 

118MB 

0s 


2 




13s 

3s 

25 

48 

49 

118MB 

0s 


3 




12s 

3s 

24 

46 

47 

118MB 

0s 

ASW 

1 

55 

0 

0 

300s 

10s 

- 

- 

- 

- 

TimeOut 

Apollo 

1 

269 

0 

0 

171s 

59s 

1677 

26714 

26716 

2130MB 

6058s 


2 

245 

26 

2 

3554s 

560s 

2756 

43984 

43986 

2120MB 

10869s 


3 

215 

56 

2 

5850s 

1371s 

2262 

34570 

34572 

2181MB 

8083s 


4 

187 

84 

2 

4807s 

2036s 

- 

- 

- 

- 

TimeOut 


Table 2: Experimental results for iDiscovery with both optimizations 


Subjects 

Iter. 

| Invariants 

[ Invariant Cost 

) Symbolic Execution Cost 



Num 

Del 

New 

Tracing 

Inference 

Tests 

SCalls 

States 

Memory 

Time 

TCAS-vl 

1 

207 

mm 


40s 

11s 

178 

3538 

20835 

121MB 

45s 


2 

59 



106s 

12s 

72 

2072 

8353 

120MB 

24s 

TCAS-v2 

1 

208 

0 

0 

100s 

13s 

279 

7644 

50143 


93s 


2 

59 

149 

0 

166s 

14s 

172 

4656 

20137 


51s 

TCAS-v3 

1 

149 

0 


218s 

16s 

447 

11562 

82323 

120MB 

129s 


2 

59 

90 


265s 

20s 

372 

8940 

42821 

119MB 

91s 

WBS 

1 

15 

0 

0 

12s 

2s 

28 

94 

539 

118MB 

Is 


2 

12 

4 

1 

14s 

3s 

25 

48 

441 

118MB 

0s 


3 

11 

4 

0 

12s 

3s 

24 

46 

437 

118MB 

0s 

ASW 

1 

55 

0 

0 

301s 

10s 


4809 

44154 

2080MB 

843s 


2 

42 

13 

0 

316s 

15s 


4580 

36857 

2091MB 

802s 

Apollo 


269 

0 

0 

159s 

56s 

82 

704 

802 

1105MB 

34s 



213 

58 

2 

161s 

79s 

35 

303 

401 

1718MB 

17s 



185 

86 

2 

68s 

88s 

14 

470 

716 

763MB 

143s 



175 

96 

2 

27s 

91s 

0 

348 

1394 

898MB 

135s 


nates, it takes iDiscovery at most four iterations to reach 
the fix-point. Since iDiscovery without optimizations does 
not finish within the 20-hour limit on ASW and Apollo, 
optimizations to mitigate the path explosion problem are 
needed. 

The results in Table 2 demonstrate that the iDiscovery op- 
timizations are able to reduce the symbolic execution cost 
dramatically. It takes more than one hour to complete sym- 
bolic execution on Apollo instrumented with assertions in 
the first iteration of iDiscovery without optimizations, while 
symbolic execution is completed in 34 seconds when iDis- 
covery is used with optimizations, indicating a speed-up of 
more than 100X. Note that the number of symbolic states 
may increase when using optimizations because of the addi- 
tional non-determinism choice points added by SPF at the 
Verify modeling primitives. For example, for TCAS-v3, the 
first iteration of symbolic execution without optimizations 
explores 14805 states, while the first iteration of symbolic 
execution with optimizations explores 82323 states. How- 
ever, the symbolic execution time is much less for the latter, 
because a large number of the symbolic states are pruned by 
the iDiscovery optimizations. The optimizations not only re- 
duce the total symbolic execution time at each iteration of 
iDiscovery, but also reduce the number of iterations for some 


subjects as seen in Table 2. For all three versions of TCAS, 
iDiscovery with no optimizations requires four iterations to 
reach a fix-point, while iDiscovery with optimizations re- 
quires only two iterations to reach the same fix-point. The 
reason is that the first optimization (assertion separation) 
greatly simplifies path constraint complexity and can solve 
more constraints within the solver time limits for each con- 
straint, thus leaving less solvable constraints for following 
iterations, and reducing the number of iterations. 

Different iDiscovery Optimizations. To better under- 
stand the effects of the optimizations, we evaluate iDiscovery 
with each optimization independently of the other on all of 
the artifacts. The assertion separation optimization is de- 
noted as Optimization- 1, and the violation restriction opti- 
mization is denoted as Optimization-2. The invariants gen- 
erated on the last iteration of Optimization- 1, Optimization- 
2, iDiscovery with no optimizations and iDiscovery with 
both optimizations are the same. Thus, in Table 3, we show 
only the symbolic execution cost for each of the iDiscovery 
configurations to study the impact of each individual opti- 
mization. In the table, Columns 1 and 2 list the correspond- 
ing artifacts and iterations; Columns 3 to 5, 6-8, 9-11, and 
12-14 list the symbolic execution cost for each iteration of 
iDiscovery with no optimizations, with only Optimization- 
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Table 3: Symbolic execution costs for iDiscovery using different optimizations 


Subjects 

Iter. 

No Optimizations 

Optimization- 1 

Optimization-2 

Both Optimizations 



SCalls 

States 

Time 

SCalls 

States 

Time 

SCalls 

States 

Time 

SCalls 

States 

Time 

TCAS-vl 

1 

10066 

10067 

250s 

12044 

29341 

201s 

1454 

1455 

25s 

3538 

20835 

45s 


2 

2894 

2895 

44s 

2312 

8593 

25s 

2934 

2935 

40s 

2072 

8353 

24s 


3 

2342 

2343 

28s 

- 

- 

- 

2202 

2203 

25s 

- 

- 

- 


4 

2090 

2091 

22s 

- 

- 

- 

1950 

1951 

21s 

- 

- 

- 

TCAS-v2 

1 

13944 

13945 

348s 

23866 

56561 

406s 

18814 

18815 

380s 

7644 

50143 

93s 


2 

5604 

5605 

93s 

5298 

20779 

59s 

6860 

6861 

84s 

4656 

20137 

51s 


3 

5404 

5405 

64s 

- 

- 

- 

5184 

5185 

54s 

- 

- 

- 


4 

4752 

4753 

50s 

- 

- 

- 

4532 

4533 

43s 

- 

- 

- 

TCAS-v3 

1 

14804 

14805 

356s 

53116 

123877 

859s 

24316 

24317 

508s 

11562 

82323 

129s 


2 

5830 

5831 

87s 

11068 

44949 

109s 

7720 

7721 

99s 

8940 

42821 

91s 


3 

5750 

5751 

63s 

- 

- 

- 

5470 

5471 

54s 

- 

- 

- 


4 

5022 

5023 

46s 

- 

- 

- 

4742 

4743 

41s 

- 

- 

- 

WBS 

1 

100 

101 

0s 

100 

545 

0s 

100 

101 

0s 

94 

539 

Is 


2 

48 

49 

0s 

48 

441 

0s 

48 

49 

0s 

48 

441 

0s 


3 

46 

47 

0s 

46 

437 

0s 

46 

47 

0s 

46 

437 

0s 

ASW 

1 

- 

- 

TimeOut 

60861 

100206 

11970s 

112322 

112323 

32028s 

4809 

44154 

843s 


2 

- 

- 

- 

4580 

36857 

770s 

4571 

4572 

773s 

4580 

36857 

802s 

Apollo 

1 

26714 

26716 

6058s 

704 

802 

33s 

12259 

12261 

8550s 

704 

802 

34s 


2 

43984 

43986 

10869s 

303 

401 

16s 

17201 

17203 

14771s 

303 

401 

17s 


3 

34570 

34572 

8083s 

470 

716 

143s 

16221 

16223 

14066s 

470 

716 

143s 


4 

- 

- 

TimeOut 

348 

1394 

136s 

14548 

14550 

12884s 

348 

1394 

135s 



(a) TCAS-vl 


(b) TCAS-v2 


(c) TCAS-v3 


Figure 5: Symbolic execution costs (in seconds) for iDiscovery with/without Green 


1, with only Optimization-2, and with both optimizations, 
respectively. Based on the results, we make the following 
observations. First, Optimization- 1 can reduce the sym- 
bolic execution time as well as the number of iterations. To 
illustrate, for TCAS-vl, iDiscovery without optimizations 
requires four iterations to finalize the invariants with a to- 
tal symbolic execution time of 5 minutes 44 seconds, while 
Optimization- 1 requires only two iterations and 3 minutes 
46 seconds. Second, Optimization-2 reduces the symbolic 
execution time without reducing the number of iterations. 
The reason is that Optimization-2 only restricts the number 
of violations for each assertion, but does not separate the in- 
teraction between different assertions. Third, the combina- 
tion of the two optimizations yields much better results than 
either of the optimizations independently. To illustrate, at 
the end of the first iteration of ASW, the symbolic execution 
time is more than three hours for Optimization- 1 and more 
than eight hours for Optimization-2 (the time for iDiscov- 
ery with no optimization is more than 20 hours). However, 
symbolic execution for iDiscovery with both optimizations 
finishes within 15 minutes, indicating a speed-up of more 
than 14X and 37X over Optimization- 1 and Optimization- 

2, respectively. These results emphasize the importance of 
reducing the complexity of assertions in each program path 
(addressed by Optimization- 1) and reducing the number of 
times an assertion is checked across different program paths 
(addressed by Optimization-2). 

Different Solver Configurations. We also study how 
state-of-the-art solver techniques based on constraint re-use 
can benefit iDiscovery. Figure 5 shows the symbolic execu- 


tion time for each iteration of iDiscovery with and without 
the Green solver interface, which reuses constraint solution 
histories from previously observed executions. Note that we 
reset the Green solver at the beginning of each experiment 
on each artifact. The reductions can be even more dramatic 
if we allow for constraint re-use across different programs. 
Each sub-figure in Figure 5 represents the results for one 
subject. In each figure, each stacked column denotes the 
total symbolic execution time (in seconds) for each of the 
four configurations (i.e. , no optimizations with no green, no 
optimizations with green, optimizations with no green, and 
optimizations with green). In each stacked column, the four 
different colors show the time cost distribution for differ- 
ent iterations. Note that we do not show Green results for 
WBS because its symbolic execution cost is too small to in- 
vestigate (Os for the majority cases), and we do not show 
Green results for ASW and Apollo because the current ver- 
sion of Green does not support the CORAL solver which is 
required for solving invariant constraints of these artifacts. 
From the results in Figure 5, we make the following observa- 
tions. First, the use of the Green solver extension can reduce 
the cost of symbolic execution independent of the optimiza- 
tions. Second, using the Green extension together with the 
optimizations can yield even better results. To illustrate, 
consider the first iteration of TCAS-vl; Green is able to re- 
duce the symbolic execution cost of iDiscovery without the 
optimizations from 4 minutes and 10 seconds to 2 minutes 
and 46 seconds. However, when optimizations are also used, 
symbolic execution takes just 8 seconds, which is much bet- 
ter than applying them separately. Thus, we believe that the 
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combination of a Green-like solver extension and our opti- 
mizations can make future applications of iDiscovery even 
more tractable. 

Different Initial Test Suites. In practice, it is possible for 
the test suites provided to iDiscovery to be generated using 
different test generation techniques, and for the test suites to 
vary in size and coverage. Thus, we also study the impact of 
test suites generated by different techniques and test suites 
of various sizes on the performance of iDiscovery. As we al- 
ready studied the performance of iDiscovery when given an 
initial test suite generated by symbolic execution, we now 
analyze the performance of iDiscovery when given an initial 
test suite generated randomly. We randomly generated 5%, 
10%, 15%, 20%, and 100% of the number of initial tests gen- 
erated by symbolic execution as initial test suites. We then 
apply iDiscovery with optimizations on each initial test suite 
for each artifact. The final set of candidate invariants and 
the time cost for the various initial tests are quite similar. 
However, the number of added (augmented) invariants and 
refuted invariants differs. Therefore, we show the number 
of added invariants and refuted invariants for each artifact 
and initial suite combination in Figure 6. In Figure 6a, the 
horizontal axis shows the different initial test suites ( symbc 
denotes the symbolically generated test suites, rand denotes 
the randomly generated test suites, and x%-rand denotes 
x% of the randomly generated test suites); the vertical axis 
shows the number of refuted invariants by iDiscovery for 
a given initial test suite; each line show the corresponding 
results for each artifact. Similarly, Figure 6b shows the num- 
ber of added invariants by iDiscovery for each artifact and 
each type of initial test suite. To understand the differences 
in performance of iDiscovery for different initial test suites, 
we also show the instruction coverage, branch coverage, and 
line coverage for each initial test suite for each subject in Fig- 
ure 7. For example, in Figure 7a, the horizontal axis shows 
the different initial test suites, while the vertical axis shows 
the instruction coverage for each suite on each artifact. 

Based on the results, we highlight several interesting find- 
ings. First, the number of refuted invariants is low for most 
artifacts when initial test suites have extremely low or high 
coverage. The reason is that when the initial test suites 
have extremely low coverage, the number of invariants in- 
ferred by the first iteration of iDiscovery is limited by the 
small set of trace samples, making the number of candidate 
invariants to be refuted small; when the initial test suites 
have extremely high coverage, the number of invariants in- 
ferred during the first iteration is also limited, because the 
high-coverage test suites can already refute a number of in- 
correct or imprecise invariants. Second, for each subject, the 
number of augmented invariants consistently goes down as 
the initial test suite coverage goes up. The reason is that 
traditional dynamic invariant inference techniques, such as 
Daikon, first generate candidate invariants for the covered 
program behaviors, and then try to refute them based on 
additional tests. Therefore, when a test suite has high cov- 
erage, Daikon has likely already computed the largest possi- 
ble set of invariants (with potentially many false- p osit ives ) , 
so that iDiscovery can only help refute incorrect /imprecise 
invariants (but cannot augment correct /precise invariants). 

5.4 Threats to Validity 

Threats to construct validity. The main threat to con- 
struct validity for our study is the set of metrics used to 


evaluate the different iDiscovery configurations. To reduce 
this threat, we use the number of finally generated candidate 
invariants, augmented invariants, as well as refuted invari- 
ants to measure the effectiveness of iDiscovery; we also use 
test tracing time, invariant inference time, and symbolic ex- 
ecution time of each iteration of iDiscovery to measure the 
efficiency of iDiscovery. Our study still has a threat to con- 
struct validity: although all the refuted invariants by iDis- 
covery are proven to be incorrect or imprecise, we did not 
check the quality of the augmented invariants. 

Threats to internal validity. The main threat to inter- 
nal validity is the potential faults in the implementation of 
different configurations of iDiscovery or in our data analysis. 
To reduce this threat, the first author carefully reviewed all 
the code for iDiscovery and data analysis during the study. 
Threats to external validity. The main threat to exter- 
nal validity is that our experimental results may not general- 
ize to other contexts, including programs, initial test suites, 
invariant inference tools, and constraint solvers. Further re- 
duction of threats to external validity requires more exten- 
sive evaluation on more invariant inference tools and con- 
straint solvers, as well as more and larger subject programs. 

6. RELATED WORK 

Automated techniques for discovering invariants can gen- 
erally be categorized as either a static or a dynamic inference 
technique. Static inference techniques are typically sound, 
but in practice, suffer from several limitations including the 
undecidability of the underlying mathematical domains, and 
the high cost of modeling program states. Much of the static 
inference work [5,22,32] uses abstract interpretation or sym- 
bolic execution to approximate the fix-point of properties on 
semantics of program expressions. Some work constructs, 
derives, or strengthens specifications by using existing spec- 
ifications [2,10,15,18,27,34]. While several projects [13,25] 
have used static verification tools, e.g., ESC/ Java [14] to 
check the invariants, the results are not used as feedback to 
further improve invariant discovery as is done in this work. 

Dynamic invariant detection techniques follow two main 
approaches. They either start with a set of candidate invari- 
ants and refine them using program trace information from 
a user-provided test suite [12,16], or they use the trace infor- 
mation to essentially build the invariants “from scratch” [9] . 
In this work, we use Daikon, which follows the first approach. 
Daikon first instantiates a set of initial candidate invariants 
based on a database of invariant templates, and then during 
execution of a test suite, it tracks the values of program vari- 
ables and uses these values to refute invariants in the initial 
set [12]. Whereas, a tool such as DySy takes the second ap- 
proach and builds the invariants by collecting symbolic path 
conditions while executing the user-supplied test suite [9]. 
DySy then summarizes the generated path conditions as in- 
variants. Note that unlike symbolic execution techniques, 
DySy is not reliant on a constraint solver or a decision pro- 
cedure to check feasibility of the invariants since it relies 
exclusively on concrete execution traces. A key difference 
between the two approaches is that techniques starting with 
candidate invariants, such as Daikon, may over-approximate 
the inferred invariants if the set of test traces does not pro- 
vide enough information to refute the generated candidate 
invariants. On the contrary, techniques that build invariants 
“from scratch”, such as DySy, can generate a more precise set 
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Figure 6: The trends for added and refuted invariants for different initial test suites 



of invariants, but the invariants are more similar to symbolic 
summaries than the invariants generated by Daikon. 

The number of invariants generated by dynamic invari- 
ant detection techniques may be quite large, and the quality 
of the generated candidate invariants for techniques such 
as Daikon [12] and DIDUCE [16], depends largely on the 
collection of invariants in its invariant repository. Recent 
studies have also shown that the test suite used to generate 
the program traces can affect the quantity and quality of 
the generated candidate invariants [12,28]. Recently, Li et 
al. [21] presented a technique to help derive more relevant in- 
variants and reduce noise. Their technique enhances the dy- 
namic invariant detection algorithm by including additional 
(“secondary”) constraints which relate classes of invariants. 
These constraints can also be inferred automatically through 
dynamic observations of program behavior. This work is or- 
thogonal to iDiscovery and would be interesting to explore 
in conjunction with iDiscovery in future work. 

The work most closely related to iDiscovery is that of 
Xie and Notkin [35]. Their approach enhances both tests 
and specifications using a feedback loop between dynamic 
invariant discovery using Daikon and test generation using 
Jtest [7]. This is similar to the feedback loop in iDiscovery, 
however, the approach in [35] uses the specifications com- 
puted by Daikon to guide the test generation process and the 
process includes a test selection step which involves manual 
inspection of the generated tests. The work presented here 
is fully automated and uses a (bounded) symbolic execu- 
tion of the program instrumented with the candidate invari- 
ants to compute additional test inputs. Howar et al. [17] 
proposed an iterative learning algorithm, X-PSYCO, which 
combines dynamic analysis and symbolic component anal- 
ysis to infer constraints on method parameters, i.e. , invari- 
ants. Unlike the method- level pre- and post-conditions gen- 
erated by Daikon, the results of X-PSYCO are represented 
using finite-state automata and specify a component’s inter- 
face in terms of safe method sequences where the inferred 


constraints on method parameters serve as guards on the 
transitions in the automata. 

Our recent work on iProperty [36] introduces an approach 
for incremental checking of conformance of code to specifica- 
tions that evolve. Specifically, iProperty introduces a prop- 
erty differencing algorithm that computes logical differences 
between two sets of properties to allow checking code against 
a minimal set of properties taking into account the previous 
version of code and properties already checked as well as 
those checking results (e.g., any counterexamples that were 
previously found) . Property differencing of iProperty can in 
principle benefit iDiscovery by further optimizing its itera- 
tive use of symbolic execution in enhancing the quality of 
invariants discovered. We plan to investigate the combina- 
tion of iProperty and iDiscovery in the future. 

7. CONCLUSION 

This paper introduced iDiscovery, a novel technique that 
applies two well-known approaches - dynamic invariant dis- 
covery and symbolic execution - in synergy to discover higher 
quality invariants. The Daikon tool for invariant discovery 
and the Symbolic PathFinder (SPF) tool for symbolic ex- 
ecution provide the enabling technology. A feedback loop 
iteratively runs Daikon and SPF to discover more accurate 
and complete invariants until a fix-point is reached. Two 
optimizations, namely assertion separation and violation re- 
striction, enhance the efficiency of iDiscovery by reducing 
the cost of symbolic execution. An experimental evaluation 
using a suite of Java subject programs shows that iDiscov- 
ery converges to a set of higher quality invariants than the 
initial set in a few iterations. 
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